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ABSTRACT 

Virtual  environments  for  command  and  control  typically  represent  spatial  information  about  entities,  such  as 
the  location  of  friendly  and  enemy  forces.  Iconography  and  interaction  standards  for  such  displays  are  well 
defined.  Less  attention  has  been  paid  to  representing  non-geographical  information,  such  as  information 
about  the  state  of  knowledge  and  decision  making  in  a  command  staff.  We  report  here  on  two  technologies 
designed  to  enable  commanders  to  measure,  monitor,  and  manage  knowledge  and  decision  making.  The 
IMAGES  tool  (now  under  development)  exploits  communication  and  language  analysis  technologies  as  well 
as  network  visualization  techniques  to  help  commanders  explore  the  distribution  of  knowledge  in  written 
communications  (e.g.,  chat  and  email).  The  CENTER  tool  (now  implemented)  exploits  a  theory  of 
collaborative  critical  thinking  for  representing  the  state  of  decision  making  within  distributed  teams.  Here,  we 
describe  challenge,  requirements,  and  two  solutions  for  measuring,  monitoring,  and  managing  knowledge 
using  virtual  environments  in  command  and  control  operations. 


INTRODUCTION 

Modem  Command  and  Control  (C2)  systems  harness  geographical  databases,  imagery,  other  sensor  data,  and 
standardized  symbology  to  represent  critical  information  spatially  for  C2  operations  (e.g.,  MCS,  CPoF,  and 
FBCB2)1.  Miller  (2005)  has  noted  that  computerized  maps  are  necessary  for  C2,  but  are  inevitably  deceptive; 
unambiguous  symbology  masks  identification  errors,  severe  latencies,  and  systematic  omission  of  data.  He 
explains  that,  “There  is  often  an  assumption  that  the  symbology  or  data  on  a  map  is  completely  accurate.  A 
map’s  symbology  must  distort  the  data  in  an  effort  to  present  a  coherent  informative  picture...  [and]  the  data 
behind  that  visual  representation  is  distorted  as  well.”  Curiously,  mapping  systems  offer  little  support  for 
identifying  and  resolving  problematic  data.  For  example,  maps  do  not  point  to  the  people  with  knowledge 
about  the  data,  its  flaws,  and  its  impact  on  decisions.  Information  systems,  in  sum,  are  not  well  integrated  with 
the  social  system  of  cartographers,  geologists,  multi-source  analysts,  correlation  analysts,  and  others  who 
interpret  data  and  diagnose  data  problems  for  decision  makers. 

Our  objective  is  to  integrate  data  systems  with  social  systems.  To  do  this,  we  are  developing  tools  that  help 
leaders  to  measure,  monitor,  and  manage  the  knowledge  of  team  members. 
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We  are  concerned  with  two  aspects  of  knowledge:  (1)  the  distribution  of  information  across  the  team  or 
organization  and  (2)  team  judgments  about  the  quality  of  information  and  decision  making  based  on  it. 
Technology  that  addresses  these  aspects  of  knowledge  well  would  create  a  Common  Operational  Picture  of 
Team  Knowledge  (COPTK).  Such  technology  is  not  common;  we  are  not  aware  of  any  extant  example  of  a 
COPTK.  Such  technology  is  technologically  feasible.  Knowledge  and  judgments  on  knowledge  and  decisions 
are  correctly  viewed  as  cognitive  states;  and  from  this  perspective  they  are  difficult  to  assess.  Laboratory 
measurement  of  cognition  involves  highly  artificial  tasks  and  intrusive  instrumentation  (e.g.,  eye  tracking, 
neuroimaging).  However,  knowledge  and  judgments  are  also  products  of  collaboration  that  emerge  in  the 
natural  course  of  communication,  and  that  can  be  efficiently  expressed  in  response  to  concise  survey  items. 
Thus,  it  is  possible  to  measure  collaborative  cognition,  visualize  these  measures  so  that  commanders  can 
monitor  team  state,  and  give  leaders  the  capability  to  manage  cognitive  collaboration. 

We  are  addressing  the  challenge  of  measuring,  monitoring,  and  managing  cognitive  collaboration  with  two 
technologies  that  we  are  developing  under  the  Collaborative  Knowledge  Management  program  of  the  Office 
of  Naval  Research.  (1)  The  Instrument  for  Measuring  and  Advancing  Group  Environmental  Situational 
awareness  (IMAGES)  captures  communications,  analyzes  them, 
and  presents  data  concerning  the  distribution  of  knowledge 
across  an  organization.  (2)  Collaboration  for  ENhanced  TEam 
Reasoning  (CENTER)  captures  brief  responses  to  survey  items, 
analyzes  them,  and  presents  data  concerning  staff  judgments 
about  the  state  of  knowledge  and  decision  making.  We  describe 
these  technologies  in  detail,  below. 

IMAGES 

IMAGES  is  a  software  tool  designed  to  improve  the  distribution 
of  knowledge  within  organizations.  IMAGES  consists  of  a  repository  of  written  communications  (e.g.,  chat, 
email)  among  members  of  a  potentially  distributed  organization,  modular  (or  swappable)  tools  that  compute 
measures  of  the  distribution  of  knowledge  represented  by  those  communications,  and  a  visualization  system 
that  enables  commanders  to  easily  monitor  the  organization’s  knowledge  state.  Components  of  IMAGES  are 
variously  in  design,  development,  or  are  largely  completed. 


Function 

State  of 

Knowledge 

State  of 

Judgments 
&  Decisions 

Measure 

IMAGES 

CENTER 

Monitor 

Manage 
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Measuring  Knowledge 

The  product,  when  fully  developed,  will  support  three  forms  of 
communication  measurement:  Network  Text  Analysis,  Dynamic 
Network  Analysis,  and  Flow  Analysis. 

Network  Text  Analysis:  IMAGES  formats  communications  data  for 
semantic  analysis  by  AutoMap  (Diesner  and  Carley  2004;  2005), 
developed  by  our  partners  at  Carnegie  Mellon  University.  Automap 
standardizes  terms  using  a  mission-specific  thesaurus,  filters  out  terms, 
cross-indexes  terms,  and  computes  counts  of  the  frequency  and  co¬ 
occurrence  of  terms.  These  counts,  which  represent  the  knowledge  in  a 
communication,  can  then  be  compared  between  teams.  For  example,  we 
can  measure  the  knowledge  that  is  held  in  common  between  two 
individuals  or  teams  by  computing  the  proportion  of  knowledge  they 
both  express  (the  intersection  of  knowledge)  to  the  knowledge  that 
either  expresses  (the  union;  see  Figure  1).  Preliminary,  experimental 
research  has  validated  this  measurement  strategy  in  a  small  team  setting 
(Weil,  Carley,  Diesner,  Freeman,  &  Cooke,  2006). 

Dynamic  Network  Analysis  Metrics :  A  second  class  of  metrics 
leverages  the  science  of  dynamic  network  analysis  developed  at  CMU  and  the  associated  meta-matrix 
approach  to  relational  data  (Carley  2003;  Carley  and  Reminga  2004).  The  meta-matrix  specifies  several 
classes  of  data  that  exist  in  communications,  specifically  data  about  people,  knowledge,  resources,  tasks,  and 
organizations.  Crossing  these  data  types  produces  useful  input  for  network  analyses.  For  example,  connections 
derived  from  personnel  data  alone  generate  a  social  network  (“Who  talks  to  whom”).  Crossing  personnel  and 
knowledge  data  generates  a  knowledge  state  network  (“Who  knows  what”).  Crossing  knowledge  data 
produces  an  information  network  (“What  is  relevant  to  what”).  These  are  the  networks  of  primary  interest  in 
IMAGES  at  the  present  time.  We  can  apply  the  range  of  graph  theoretic  social  network  metrics  to  these 
networks  to  discern,  for  example,  which  people  or  concepts  are  central  to  an  organization’s  discourse,  and 
which  are  gateways  to  different  clusters  to  other  groups  or  concepts. 

Flow  analysis  metrics:  Patterns  in  the  occurrence,  sequence,  and  timing  of  communications  can  be  identified 
and  assessed  using  measures  developed  by  our  partners  at  Arizona  State  University  (Cooke,  Salas,  Cannon- 
Bowers  &  Stout  2000;  Cooke  2005),  and  referred  to  as  FAUCET  (Flow  Analysis  of  Utterance 
Communication  Events  in  Teams).  These  are  content-independent  measures,  and  -  though  they  assess  only 
who  communicates,  when,  and  in  what  order  -  scores  on  these  measures  have  been  experimentally  shown  to 
be  strongly  correlated  with  effective  team  performance  and  team  situational  awareness  in  domains  such  as 
UAV  operations. 

Monitoring  Knowledge 

The  conceptual  design  for  data  visualization  in  IMAGES  (see  Figure  2)  illustrates  functionality  with  which 
users  will  configure  communications  analyses,  explore  analysis  results,  and  review  diagnoses  and 
recommended  actions.  A  sub-set  of  the  illustrated  features  is  currently  being  implemented. 


Figure  1:  Knowledge  held  in 
common  as  a  proportion  of  all 
knowledge:  AflB  /  AUB. 
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Figure  2:  The  IMAGES  concept  interface  provides  controls  for  the  analysis  of  communications  (left  pane),  a 
visualization  of  communications  frequency  (top  center)  and  topics  (bottom  center),  and  action  recommendations. 

In  the  left  hand  pane  of  Figure  2,  the  operator  is  able  to  configure  the  tool,  specify  the  types  of  data  sources 
with  which  to  create  a  semantic  network,  and  set  the  timeframe  for  analysis.  Both  formal  (reports  and  briefs) 
and  informal  data  sources  (chat,  e-mail)  might  be  selected  for  analysis,  assuming  that  these  are  available  in  the 
collaborative  environment.  These  parameters  determine  how  communications  is  measured  in  IMAGES.  The 
top-center  pane  presents  a  network.  In  this  case,  the  nodes  -  various  Joint  Forces  components,  Coalition 
Forces,  and  Non-Governmental  Organizations  -  represent  organizations  involved  in  a  Coalition  relief  effort. 
The  connections  between  nodes  indicate  communications  volume;  the  absence  of  an  arc  between  nodes 
indicates  there  was  no  direct  communication,  while  a  thin  line  indicates  limited  communication.  In  the  bottom 
center  of  the  panes,  a  graph  indicates  the  topics  being  discussed  (e.g.,  administration,  operations,  logistics), 
based  on  several  generic  thesauri.  In  this  case,  the  topics  correlate  with  the  assignments  of  major  staff 
organizations  in  a  typical  military  organization. 

Using  IMAGES,  an  operator  could  monitor  whether  the  appropriate  organizations  are: 

1 .  Talking  to  each  other,  by  noting  the  connections  between  nodes; 

2.  Talking  about  the  appropriate  topics,  by  checking  the  topic  profile  ; 

3.  Talking  about  them  in  the  same  contexts,  by  observing  the  darkness  of  the  bar. 

The  ability  to  manipulate  the  time  frames  involved,  the  communication  mediums  used,  and  the  level  of  detail 
desired  could  afford  greater  flexibility  to  the  operator  in  a  future  implementation  of  IMAGES. 

Managing  Knowledge 

The  right  hand  pane  of  Figure  2  illustrates  one  notion  for  a  management  aid  within  IMAGES.  It  presents  an 
automated  diagnosis  of  a  knowledge  state  (e.g.,  “Units  A  and  C  are  not  communicating  at  all.  Units  A  and  B 


2-4 


RTO-MP-HFM-136 


dMNATO 
WP  I  OTAN 


Measuring,  Monitoring,  and  Managing 
Knowledge  in  Command  and  Control  Organizations 


are  not  discussing  enemy  position”).  This  diagnosis  is  computed  by  comparing  the  structure  and  content  of  an 
observed  knowledge  state  to  that  of  a  model  or  normative  knowledge  state.  The  normative  knowledge  state 
might  be  defined  directly  by  specifying  the  anticipated  topics  of  communication  between  team  members. 
Alternatively,  it  might  be  inferred  from  a  computational  model  of  the  task  assignments  of  team  members  and 
the  information  and  communication  requirements  between  tasks  (e.g.,  “The  executor  of  Task  A  must  pass 
weather  data  to  the  executor  of  Task  B”).  In  addition  to  diagnosis,  the  interface  presents  a  remedy,  such  as 
ordering  team  members  to  review  some  aspect  of  a  mission  which  should  be  -  but  has  not  yet  been  -  a  topic 
of  discussion. 

CENTER 

CENTER  is  a  software  system  designed  to  improve  collaborative  critical  thinking  by  teams  about  knowledge 
and  decisions.  The  CENTER  tool  (1)  enables  a  leader  to  query  members  of  the  organization  concerning  the 
state  of  mission  knowledge  and  decisions,  (2)  elicits  brief  responses  and  summarizes  them  statistically,  and  (3) 
presents  these  measures  to  leaders  with  guidance  concerning  the  issues  on  which  leaders  should  focus  their 
attention  and  that  of  members.  In  short,  CENTER  helps  leaders  to  measure,  monitor,  and  manage 
collaborative  critical  thinking  about  team  knowledge  and  decisions.  CENTER  is  a  fully  implemented  software 
package  that  runs  on  networked  Windows  and  Linux  platforms. 

Measuring  Knowledge 

Collaborative  critical  thinking  involves  interaction  between  team  members  to  reveal  uncertainty  concerning 
knowledge  or  decisions,  identify  its  sources,  and  devise  ways  to  diminish  it  or  accommodate  it.  These 
collaborative  activities  may  help  team  members  improve  estimates  of  risk  and  refine  plans  to  accomplish 
missions  in  the  face  of  risk. 

Our  definition  of  this  collaborative  activity  derives  from  the  literature  on  individual  critical  thinking, 
specifically  the  work  of  Cohen,  et  al.  (1998;  1997),  which  empirically  validated  that  individuals  who  engage 
in  several  specific  critical  thinking  behaviors  outperform  others  on  tactical  assessment,  planning,  and  decision 
making  tasks  Collaborative  critical  thinking  leverages  this  set  of  behaviors  in  its  definition  and  measures. 
CCT  consists  of  (1)  monitoring  interactions  that  alert  other  team  members  to  the  existence  of  uncertainty,  (2) 
assessment  interactions  in  which  team  members  evaluate  the  opportunity  (e.g.,  available  time)  and  need  (e.g., 
priority  or  stakes)  to  resolve  the  uncertainty,  (3)  critiquing  interactions  in  which  team  members  identify  the 
source  of  uncertainty  (specifically:  gaps  in  knowledge,  conflicting  interpretations,  and  untested  assumptions), 
and  (4)  action  planning  and  execution  activities  that  resolve  problems  with  knowledge  or  decisions,  or  that 
compensate  for  irresolvable  uncertainty. 

Collaborative  Critical  Thinking  is  frequently  applied  to  two  objects.  A  focus  on  the  mission  involves 
critiquing  assessments  (e.g.,  of  enemy  intent,  or  the  state  of  own  forces)  and  plans  (as  is  done  in  Course  of 
Action  development  and  assessment).  A  focus  on  the  team’s  process  involves  a  critique  of  goals,  the  plans  (or 
strategies)  for  achieving  those  goals,  and  the  state  of  tasks  that  constitute  the  plan. 

We  have  crossed  these  two  dimensions  -  the  collaborative  activities  that  constitute  critical  thinking,  and  the 
objects  to  which  it  is  applied  -  to  define  the  space  of  behavior  that  must  be  measured  (see  Table  1). 
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Table  1:  CENTER  measures  critical  thinking  activities  applied  to  team  products  and  processes. 


Objects  of  CCT 

Team  products 

Team  processes 

CCT  Behavior 

Assessment 

COA/Plan 

Goals 

Plans 

Tasks 

Monitoring 

Al 

T1 

Assessment 

C2 

Critiquing  for  gaps 

T3 

Critiquing  for  assumptions 

P4 

Critiquing  for  ambiguity 

T5 

Action 

A6 

G6 

We  have  defined  measures  within  this  space  that  are  ratings  of  agreement  (strongly  disagree  to  strongly  agree) 
with  assertions  (below)  that  can  be  customized  (in  the  bracketed,  cloze  fields)  for  a  given  mission.  For 
example,  probes  relevant  to  the  marked  cells  in  Table  1  are: 

Al:  The  team's  assessment  [of _ ]  is  correct. 

A6:  The  team  is  taking  actions  to  resolve  problems  with  the  assessment  [concerning _ ]. 

C2:  The  team  has  time  to  critique  and  refine  the  plan  [regarding _ ]. 

P4:  The  team  has  identified  key  assumptions  that  have  yet  to  be  tested  concerning  its  strategy  [for _ ]. 

G6:  The  actions  of  team  members  are  consistent  with  the  mission  goals  [concerning _ ]. 

T1 :  Team  members  seek  feedback  on  their  tasks  [concerning _ ]. 

T3:  The  team  is  completing  all  tasks  [concerning _ ]. 

T5:  Team  members  seek  to  resolve  ambiguity  in  task  assignments. 

CENTER  allows  a  leader  or  facilitator  to  select  a  subset  of  probes  appropriate  to  the  mission  at  hand,  and 
customize  them  or  use  them  in  their  generic  form.  The  facilitator  can  then  trigger  the  delivery  of  each  probe  to 
networked  members  of  the  team.  Each  probe  appears  in  a  small  window  in  a  member’s  workspace  (see  Figure 
3)  with  a  rating  scale  and  two  buttons:  one  to  add  textual  comments  and  one  to  send  the  response  back  to  the 
facilitator.  Each  window  disappears  after  a  specified  period,  and  a  countdown  reminds  the  team  member  of 
this. 


In  experimental  research  conducted  with  our  colleague  Michael  Coovert,  Ph.D.,  at  the  University  of  South 
Florida,  participants  found  these  probes  to  be  useful  in  a  teamwork  exercise  executed,  and  non-disruptive  of 
taskwork  if  they  occurred  at  least  three  minutes  apart. 


u 


CE.NTE.K  probe  Window 


The  team  has  a  shared  understanding  of  the  terrorist  situation. 
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Strongly 
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Somewhat 

Disagree 

Somewhat 

Agree 

Agree 

Strongly 

Agree 

Don't  Know/ 
Does  Not  Apply 

253  seconds 

Send 

Add  Comment 

Figure  3:  CENTER  probes  are  simple,  unobtrusive,  and  rapidly  addressed. 

Monitoring  Knowledge 

CENTER  converts  responses  to  each  probe  into  numeric  values,  and  summarizes  them  as  a  mean  and  range 
(see  Figure  4).  Leaders  or  facilitators  can  view  the  numeric  responses  to  all  probes,  or  drill  down  to  inspect 
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the  responses  -  both  ratings  and  comments  -  to  any  one  probe.  In  this  way,  leaders  can  monitor  the 
organization’s  state  of  collaborative  critical  thinking  with  respect  to  mission-specific  issues. 


Figure  4:  Responses  to  CENTER  probes  are  summarized  as  means  and  range. 


Managing  Knowledge 

CENTER  helps  leaders  to  interpret  measures  of  collaborative  critical  thinking  and  take  action  to  improve  it.  It 
does  so  by  analyzing  distribution  patterns  in  each  response  and  presenting  guidance  to  the  leader.  For 
example,  assume  that  this  probe  -  “The  team  has  the  information  it  needs  to  plan.”  -  is  administered  to  the 
team  well  into  a  long,  mission  planning  task.  The  distribution  of  responses  across  the  team  may  have  a  high 
(positive)  mean  and  low  variance,  indicating  that  there  is  near  unanimous  agreement  with  the  probe  statement. 
In  this  case,  CENTER  advises  the  leader  as  follows: 

The  team  members  believe  that  they  have  the  information  they  need  to  plan.  Suggest  that  they  move  on.  If 
there  is  a  large  team  then  probe  for  lone  dissenters ,  if  found,  engage  them,  for  example:  "Would  you  like  to 
add  anything?  " 
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If  the  responses  across  the  team  exhibit  an  average  mean  and  high  variance  (indicating  that  some  people  agree 
with  the  probe  while  others  disagree),  CENTER  returns  the  following  guidance: 

The  team  members  do  not  agree  whether  the  team  has  the  information  it  needs  to  plan.  Seek  to  understand 
why  there  is  so  little  consensus.  Tell  those  who  disagreed  with  the  probe:  " Share  your  concerns  regarding 
insufficient  information  needed  for  planning  with  the  other  team  members.  Tell  them  what  information  seems 
to  be  missing.  See  if  they  have  that  information. "  Tell  the  team  members  who  agreed  with  the  probe  " Not  all 
team  members  believe  that  there  is  enough  information  to  plan.  Find  out  what  information  is  missing. "  Help 
the  team  quickly  get  the  information  it  needs. 

The  leader  can  then  send  messages  to  the  team  members,  or  the  leader  can  engage  all  or  a  subset  of  team 
members  in  an  instant  message  session  (see  Figure  5)  to  facilitate  more  complete  knowledge  management. 


Figure  5:  CENTER’S  Instant  Message  capability 
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CONCLUSION 

IMAGES  and  CENTER  are  designed  to  make  the  social  system  of  knowledge  more  accessible  to  leaders. 
These  technologies  measure  the  state  of  knowledge  within  teams  as  well  as  team  judgments  about  knowledge 
and  decision  making,  and  they  enable  leaders  to  monitor  and  manage  such  knowledge.  These  capabilities  are 
important  to  help  leaders  leverage  the  capabilities  of  modern  information  systems,  by  giving  them  insights 
into  the  use  of  information  from  these  systems  and  judgments  about  it.  Thus,  CENTER  and  IMAGES  may 
help  to  integrate  data  systems  with  social  systems.  Technologies  such  as  this  may  be  particularly  important  in 
distributed  organizations,  and  in  virtual  organizations2,  in  which  leaders  cannot  easily  observe  interactions  - 
such  as  “buzz”  about  specific  information  or  arguments  over  decisions  -  that  convey  the  state  of  team 
knowledge.  In  these  environments,  technology  is  needed  to  make  team  knowledge  state  accessible. 

We  invite  opportunities  to  transition  CENTER  and  IMAGES  (when  it  is  completed)  to  operational 
environments  and  to  training  environments  where  they  might  be  used  to  measure  of  teamwork  competencies 
and  support  After  Action  Reviews. 
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■  The  challenge  of  supporting 

socio-technical  systems 

■  Measuring,  monitoring,  & 
managing 

-  Knowledge 

-  Judgment 
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■  Issues 

-  Unambiguous  symbology  masks 
i.d.  errors  /  uncertainty 

-  Unambiguous  placement  masks 
latencies 

-  Systematic  omission  of  data 

■  Miller  (2005): 

-  “symbology  must  distort  the  data  in 
an  effort  to  present  a  coherent 
informative  picture” 

-  “the  data  behind  that  visual 
representation  is  distorted  as  well  ” 


Miller,  Craig  S.  (2005).  A  New  Perspective  for  the  Military: 
Looking  at  Maps  Within  Centralized  Command  and  Control 
Systems.  Air  &  Space  Power  Chronicles. 
(http://www.airpower.maxwell.af.mil/airchronicles/cc/miller. 
html) 
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Creating  a 
Socio-Technical  System 


MCS,  CPoF 
FBCB2 


Socio-Technical  system 


cartographers, 
geologists,  sensor 
specialists,  COP 
device  engineers, 
analysts... 


■  TS  SS:  Data  point  to  people  who  understand  it 

■  SS  TS:  People  reveal  their  knowledge  of  and  judgments  on  data 
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What  People  Have 


Technology 

brings 

■  Data 


People  have: 

■  Knowledge 

■  Judgment 
about  data 


Socio-Technical  systems  should  help  leaders 
Measure,  Monitor,  &  Manage 
the  knowledge  &  judgments  of 
people  about  technology  and  its  data 
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IMAGES 


Instrument  for  Measuring  and  Advancing  Group 
Environmental  Situational  Awareness 


Aptima 

Carnegie  Mellon  University:  Kathleen  Carley  &  Jana  Diesner 
Arizona  State  University:  Nancy  Cooke 


©  2006,  Aptima,  Inc. 


6 


APTIMA 

HUMAN-CENTERED 

ENGINEERING 


Objective 


■  IMAGES  will  help  leaders  answer  questions  about  the 
relations  between  &  within  knowledge  and  people 

■  Knowledge-Knowledge 

-  Have  we  “connected  the  dots”  between  Knowledge  X  &  Y? 

■  People-Knowledge 


People 

Knowledge 

People 

Social  Net 

Knowledge  Net 

Knowledge 

Information  Net 

-  Who  knows  (the  most)  about  Knowledge  X? 

-  Is  Knowledge  X  widely  shared? 

-  Which  staff  must  be  retained  to  maintain  organizational 
knowledge  of  X? 

-  Has  knowledge  changed  in  this  organization  changed  sinct 
Event  X? 

■  People-people 

-  Are  people/units  communicating  in  patterns  I  expect? 

-  Are  supervisors  communicating  with  staff? 

-  Who,  if  removed  from  my  network,  would  isolate  teams 
from  each  other? 

Carley,  K.M.,  and  Reminga,  J.  (2004).  ORA:  Organization  Risk  Analyzer.  CMU-ISRI-04- 
106.  Pittsburgh,  Carnegie-Mellon  University.  (Also  available  at:  reports- 
archive.adm.cs.cmu.edu/anon/isri2004/CMU-ISRI-04-106.pdf) 

Nancy  Cooke.  (2005).  Communication  in  Team  Cognition  .  Proceedings  of  the 
Collaboration  and  Knowledge  Management  Workshop  2005.  San  Diego,  CA. 
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Architecture 


Collaboration 

Tools 

Mail 


Chat 


IMAGES 


Analyst  User  Interface 


/^Configuration 

Interface 

Visualizations 

User  tools 
&  controls  Results 
Displays 


v~r 


Data  Processing 


Database/Coltaboratlon  Content  Repository 


User  &  Organization 
Information 
Processed  Data 


Data 

Analysis 

Toolbox 


AutoMAP 


Tool  B 
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Use  Case 


213  0  images  BE® 


File  Edit  Window  Help 


Create  mission  network  using: 

Organizational  Structure 

Combined 


Formal  Documents 

0  Briefings 
0  Reports 

Communications 

0Chat 
0  Email 
0  Radio 
0  Telephone 

From  the  last: 

Time  Span 

Days:  10  5  D 


Create  Network 


Topic  Communications:  N1 

Less  Agreement  Wore  Agreement 

Frequency 


1 ,  Administration: 

2.  Intelligence: 


3,  Operations: 

4,  Logistics: 

5,  Plans: 

6,  Communications: 

7,  Readiness  Et  Training: 

S,  Resources  &  Assessment: 


Recommendations: 


*  N1  and  N6  are  communicating  with 
each  other  moderately,  but  may 
not  have  a  shared  understanding  of 
Topic  4:  Logistics.  N1  and  N6  may 
want  to  increase  communications 
and  to  review  briefing  #05-12.  Show 
me  evidence... 

*■  JFLCC  is  communicating  heavily 
with  JFMCC  but  less  so  with  JFACC 
and  Coalition  Forces.  Please  review 
this  communication  pattern  to 
improve  situational  assessment. 
Show  me  evidence... 

*  N3  is  communicating  heavily 
regarding  Topic  2:  Intelligence,  but 
less  so  regarding  Topic  4:  Logistics 
and  Topic  3:  Operations.  Please 
review  this  pattern.  Show  me 
evidence. .. 
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Pilot  Study 


NEO:  Plan  evacuation  of 
hostage  Red  Cross  workers 

Joint  Forces 

-  Environment 

-  Intelligence 

-  Weapons 

Background,  General  & 
Specialist  Briefs 

Chat  communication  & 
reports  captured,  analyzed 


Hypotheses: 

1 .  Specialist  teams  teams  have  greater  overlap  with  briefs  in  own  expertise 
than  in  others’  ->  Not  supported 

2.  Multi-specialist  teams  have  greater  overlap  with  briefs  than  specialist 
teams  ->  Supported 

3.  Multi-specialist  teams  have  greater  overlap  with  each  other  than  the 
pairings  of  specialist  teams  ->  Supported 
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CENTER 


Collaboration  for  ENhanced  TEam  Reasoning 


Aptima 

University  of  South  Florida:  Michael  Coovert 
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Objective 


■  CENTER  helps  team  leaders  to  elicit  and  assess  the  judgments  of  members 
about  knowledge  &  manage  team  attention. 


-  Perceive  (monitor)  uncertainty,  risk 

-  Assess  opportunities  to  critique 

-  Critique  weaknesses 

■  Gaps 

■  Assumptions 

■  Ambiguity 

-  Action  taking  to  resolve  problems 
■  Issues 

-  Team  products  (assessments,  plans) 

-  Team  process 
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Example 


Probe 


-  The  team  has  the  information  it  needs  to 

■  Response  ratings 

-  Central  Mean  and  High  Variance 

■  Feedback 


plan 


I 


1 


-  The  team  members  do  not  agree  whether  the  team  has  the  information 
it  needs  to  plan.  Seek  to  understand  why  there  is  so  little  consensus  if 
time  allows.  Talk  with  those  who  disagreed.  They  may  be  (1 )  out  of 
communication  with  team  members  or  (2)  aware  of  gaps  others 
overlooked. 
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Leader  /  Facilitator 
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Use  Case 


CENTER  pr 


The  team  has  a  shared  understanding  of  the  terrorist  situation. 


r 

r 

r 

r 

C 

r 

r 

Strongly 

Disagree 

Disagree 

Somewhat 

Disagree 

Somewhat 

Agree 

Agree 

Strongly 

Agree 

Don't  Know/ 
Does  Not  Apply 

253  seconds 

Send  | 

Add  Comment 

Participant  1  ...  n 
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between  probes  (or  respondents) 
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QLNTUK  -  De, 


Event  lools 


jones 

smith 

hathaway 

sulley 

branford 

boudreaux 

hebert 

perkins 

Jackson 

kramer 

shaw 

miller 

wiese 

dunn 


Participants 


The  team  has  considered  plausible, 
alternative  plans. 


Current  Probe 


branford:  5 
shaw:  6 
Jackson:  5 
smith:  6 
dunn:  5 
wiese:  6 
miller:  6 
jones:  6 
perkins:  6 

Jackson:  5  -  now  I  got  it! 
dunn:  6  -  get  our  guy! 
wiese:  5 
kramer:  6 
branford:  6 

hebert:  6 _ 


■S  |  Graphs  I |R 


Participants'  Responses 

-00:02:02  The  team  has  a  shared  understanding  of  the  terrorist  situation. 

00:05:50  The  team  has  the  information  it  needs  to  plan. 

00:11:56  The  team  takes  action  to  resolve  problems  related  to  assessment  protocol. 
00:19:38  Team  members  understand  what  they  need  to  do. 

00:20:30  The  team  has  a  shared  understanding  of  the  terrorist  situation. 

*00:25:38  As  part  of  goal  accomplishment,  team  members  identify  working  assumptions. 
00:26:30  Alternative  plans  have  been  evaluated  relative  to  team  goals. 

00:27:16  The  team  has  considered  plausible,  alternative  plans. 


Probe  Response  Timeout:  |300  (seconds)  Set  Time  | 


Analysis  of  comments 
by  respondents  per  probe 
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■  Objective:  Integrate  social  and 
technical  systems. 

-  Technical  systems  can  point  to 
social  systems 

-  Social  systems  can  interpret 
technical  systems 

■  Measure  knowledge  &  judgment 

■  Monitor 

■  Manage  team  attention 

■  Examples 

-  IMAGES:  Knowledge 

-  CENTER:  Judgment 
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Socio-Technical  systems 
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